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Enclosure  1 


Forward 


The  PI  was  awarded  a  short  term  innovative  research  grant,  number  W91  INF-05-1-0212,  through 
the  University  of  North  Dakota.  The  grant  period  was  15  April  2005  to  14  January  2006. 
However,  during  the  grant  period  the  PI  transferred  to  Temple  University  in  August  2005.  So,  the 
PI  is  submitting  the  final  progress  report  with  the  results  that  the  PI  obtained  during  15  April  2005 
to  31  August  2005. 

The  main  objective  of  this  proposal  was  to  develop  statistical  control  theory.  Statistical 
control  is  a  generalization  of  Kalman’s  linear-quadratic-Gaussian  control,  where  we  optimize  the 
probability  density  of  the  performance  index  by  controlling  the  cumulants.  A  statistical  controller 
will  have  better  performance  and  stability  margin  than  the  linear-quadratic-Gaussian  controller.  In 
this  project,  we  investigated  the  characteristics  of  linear  statistical  controllers,  developed 
nonlinear  statistical  control  theory,  and  utilized  the  statistical  control  paradigm  in  dynamic  game 
theory. 

Although  the  linear  version  of  the  statistical  control  theory  is  almost  completely  developed, 
the  nonlinear  version  is  still  at  its  infancy.  However,  almost  all  real  systems  are  nonlinear  and 
nonlinear  control  theory  is  becoming  more  and  more  important  for  the  real  world  applications. 
This  is  especially  true  for  autonomous  systems  such  as  satellites  or  unmanned  aerial  vehicles  with 
wide  operating  conditions.  So,  the  PI  and  his  collaborators  investigated  statistical  control  for 
nonlinear  systems.  We  formulated  and  solved  a  nonlinear  statistical  control  problem  using 
dynamic  programming.  Dynamic  programming  is  a  powerful  method  that  leads  to 
Hamilton-Jacobi-Bellman  partial  differential  equations.  We  derived  the 

Hamilton-Jacobi-Bellman  equation  for  a  nonlinear  system,  and  proposed  a  method  to  solve  this 
partial  differential  equation. 

Another  interesting  research  area  related  to  statistical  control  is  dynamic  game  theory. 
Dynamic  game  theory  is  currently  an  active  and  important  research  area.  In  dynamic  game  theory, 
we  introduced  multiple  players  with  conflicting  objectives  and  optimize  cost  functions  with 
respect  to  their  objectives.  So,  this  is  an  ideal  setting  for  many  military  applications.  In  this 
project,  we  formulated  stochastic  game  problems  and  solved  them  using  statistical  control 
paradigms.  In  other  words,  we  optimized  any  function  of  the  statistical  parameters  of  the  cost 
function  instead  of  the  average  cost  as  traditionally  done  in  game  theory. 

The  development  of  statistical  control  theory  will  further  advance  the  existing  knowledge  in 
stochastic  optimal  control. 
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Statement  of  the  Problem  Studied 


In  this  project,  we  studied  characteristics  of  linear  statistical  control  and  nonlinear  statistical 
control.  We  also  studied  the  relationship  between  statistical  control  and  dynamic  game  theory.  In 
statistical  control,  we  view  the  cost  function  as  a  random  variable  and  optimize  any  of  the  cost 
cumulants  rather  than  optimizing  just  the  mean  as  in  the  traditional  linear-quadratic-Gaussian 
optimal  control.  The  summary  of  the  main  problems  studied  in  this  project  is  given  as  follows. 

LQG  control  optimizes  the  average  of  a  quadratic  cost  function  which  intuitively 
corresponds  to  the  energy  of  the  system.  In  statistical  control,  we  can  optimize  any  cumulant  of 
the  quadratic  cost  function.  Consequently  the  natural  question  to  ask  is  “what  is  the  intuitive 
meaning  behind  this  optimization?”  We  answer  this  question  using  entropy,  distribution  shaping, 
and  Bode  integral.  Also,  we  investigate  the  relations  between  cost  cumulants  and  statistical 
density. 

Even  though  many  systems  can  be  approximated  as  a  linear  system,  there  are  many 
systems  such  as  satellite  attitude  dynamics,  which  may  benefit  from  a  nonlinear  model  and  control. 
Thus,  we  proposed  to  formulate  and  solve  nonlinear  statistical  control  problems  using  dynamic 
programming  methods.  We  derived  the  Hamilton-Jacobi-Bellman  (HJB)  equation  for  n- th  cost 
moment  case.  Then  utilizing  the  relationship  between  moments  and  cumulants,  we  found  the  HJB 
equation  for  n- th  cost  cumulant  case.  Then  we  determined  the  optimal  controller  for  first,  second, 
and  third  cost  cumulant  optimization  problem  using  newly  derived  HJB  equations.  This  theory 
enabled  us  to  determine  the  optimal  statistical  controller  for  a  nonlinear  system. 

Another  theoretical  research  of  this  proposal  is  in  examining  the  relations  between 
statistical  control  and  dynamic  game  theory.  We  can  view  statistical  control  in  terms  of  dynamic 
game  theory  with  player  one  being  the  controller  and  player  two  being  the  model  error.  This  is  an 
interesting  extension  that  is  being  investigated.  On  the  other  hand,  instead  of  viewing  statistical 
control  as  a  part  of  dynamic  game  theory,  we  can  formulate  the  stochastic  game  problem  and  solve 
this  problem  using  statistical  control  concepts,  which  means  that  we  can  use  any  statistical 
parameters  (cumulants)  instead  of  using  just  the  average  cost  as  in  traditional  stochastic  game 
theory.  This  project  investigated  statistical  control  from  this  perspective.  By  applying  dynamic 
games  concepts  to  statistical  control  and  vice  versa,  we  hope  to  deepen  the  understanding  of  both 
areas. 
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Summary  of  the  Most  Important  Results 


Characteristics  of  Linear  Statistical  Control.  The  cost  function  in  stochastic  optimal  control  is 
viewed  as  a  random  variable.  Then  the  classical  linear-quadratic-Gaussian  control,  entropy 
control,  risk-sensitive  control,  and  statistical  control  can  be  viewed  as  the  cost  distribution  shaping 
methods.  We  determined  the  relations  between  entropy,  Bode  integral,  risk-sensitive  cost  function, 
and  statistical  control  cost  function.  Furthermore,  we  related  the  cost  cumulants  with  information 
theoretic  entropy,  and  Bode  integral.  The  interpretation  of  statistical  control  is  given  in  terms  of 
the  control  entropy  minimization.  We  also  related  information  theoretic  entropy  with 
exponential-of-integral  cost  function  using  a  Lagrange  multiplier  and  calculus  of  variations. 
Finally,  the  logarithmic-exponential-of-integral  cost  function  is  related  to  the  information 
theoretic  entropy  using  large  deviation  theory. 

Deterministic  Nonlinear  Statistical  Control  Theory.  A  full-state  feedback  optimal  control 
problem  is  solved  for  a  deterministic  quasi-linear  system  with  a  quasi-quadratic  cost  function. 

The  solution  method  is  based  on  algebraically  solving  Hamilton-Jacobi  equation  in  terms  of  the 
gradient  of  the  value  function.  Then  we  interpreted  the  value  function  in  terms  of  the  control 
Lyapunov  function  and  provided  the  stabilizing  controller  and  the  stability  margins.  We  derived 
an  optimal  controller  for  a  nonlinear  system  which  requires  a  solution  of  the  state  dependent 
Riccati  equation.  Moreover,  we  found  the  optimal  controller  when  the  cost  function  is  the 
exponential-of-integral  case,  which  is  risk  sensitive  control.  Then  we  showed  that  these  two 
methods  give  equivalent  optimal  controllers  for  nonlinear  deterministic  systems. 

Stochastic  Nonlinear  Statistical  Control  Theory.  A  nonlinear  stochastic  system  with 
non-quadratic  cost  function  with  respect  to  the  state  is  considered  for  the  statistical  control 
problem.  The  optimal  controller  is  determined  via  the  Hamilton-Jacobi-Bellman  equation.  The 
Hamilton-Jacobi-Bellman  equation  for  the  n-th  cost  moment  case  is  derived  as  a  necessary 
condition  for  optimality  using  dynamic  programming.  Then  the  sufficient  condition  theorem  for 
the  optimality  is  presented.  The  n-th  cost  cumulant  Hamilton-Jacobi-Bellman  equation  derivation 
procedure  is  given.  Second,  third,  and  fourth  cost  cumulant  Hamilton-Jacobi-Bellman  equations 
were  derived  using  the  proposed  procedure.  Then  the  optimizing  controller  for  the  first  and 
second  cost  cumulants  is  found  using  the  Hamilton-Jacobi-Bellman  equation.  We  transformed  the 
HJB  equation  to  a  first  order  partial  differential  equation  using  the  inversion  method.  Both  time 
varying  and  time  invariant  systems  are  considered. 

Dynamic  Game  Theory  and  Statistical  Control.  There  is  a  close  relationship  between  statistical 
control  and  dynamic  game  theory.  In  this  project,  we  investigated  the  possibilities  of  using 
statistical  control  paradigm  in  stochastic  dynamic  game  theory.  It  is  assumed  that  the  players 
will  try  to  minimize  a  linear  combination  of  cumulants  of  their  own  cost  function.  More 
specifically,  player  one  will  seek  to  minimize  a  performance  index  made  up  of  a  linear 
combination  of  cumulants  of  the  first  cost  function  and  the  player  two  will  minimize  a 
performance  index  consisting  of  a  linear  combination  of  cumulants  of  the  second  cost  function. 
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This  linear  combination  of  cumulants  could  be  different  for  each  player.  We  investigated  the 
performance  indices  involving  up  to  k-th  cost  cumulants  and  games  with  more  than  two  players. 

Statistical  Control  Meeting.  The  PI  also  held  the  meeting  with  Dr.  Michael  Sain  and  Ronald 
Diersing  at  the  University  of  North  Dakota  from  18  July  2005  to  22  July  2005.  Figure  1  shows  the 
two  researchers  in  front  of  the  Engineering  building  at  the  University  of  North  Dakota.  This  was 
proposed  in  Section  2  of  the  proposal.  During  the  meeting  we  discussed  statistical  control  issues 
and  its  relations  to  the  dynamic  game  theory.  We  discussed  the  following  issues:  Nonlinear 
Statistical  Control,  Characteristics  of  Statistical  Control,  Discrete  time  Statistical  Control,  Game 
Theoretic  Output  Feedback  Control,  Long  Term  Research,  Distribution  Shaping  Problem,  and 
Applications. 


Figure  1:  Statistical  Control  Researchers  Ronald  Diersing  and  Michael  Sain  Visiting  the 

University  of  North  Dakota 
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Publications  Listing 


The  PI  has  worked  on  the  statistical  control  theory  throughout  the  grant  period,  he  especially 
concentrated  on  the  nonlinear  statistical  control  theory.  His  paper  titled,  “Nonlinear  n-th 
Cumulant  Control  and  Hamilton-Jacobi-Bellman  Equations  for  Markov  Diffusion  Process,”  has 
been  accepted  for  publication  in  44th  IEEE  Conference  on  Decision  and  Control.  Because  the 
grant  was  delayed  for  a  few  months,  the  part  of  the  work  was  published  without  acknowledgement 
with  the  grant  number.  In  particular,  the  paper  titled,  “State-Feedback  Optimal  Controllers  for 
Deterministic  Nonlinear  Systems”  was  published  in  2005  American  Control  Conference.  Also, 
his  coworkers — Michael  Sain  and  Ron  Diersing — published  a  paper  titled,  “A  Multiobjective  Cost 
Cumulant  Control  Problem:  The  Nash  Game  Solution,”  at  the  same  2005  American  Control 
Conference  without  acknowledging  the  grant. 

(a)  Papers  Published  in  Peer-Reviewed  Conference  Proceedings 

❖  Chang-Hee  Won,  “Nonlinear  n-th  Cost  Cumulant  Control  and  Hamilton-Jacobi-Bellman 
Equations  for  Markov  Diffusion  Process,”  Proceedings  of  40th  IEEE  Conference  on 
Decision  and  Control,  Seville,  Spain,  December  2005,  accepted  for  publication. 

❖  Chang-Hee  Won,  “State-Feedback  Optimal  Controllers  for  Deterministic  Nonlinear 
Systems,”  Proceedings  American  Control  Conference,  Portland,  Oregon,  pp.  858-863, 
June  8-10,  2005. 


❖  Ron  Diersing  and  Michael  Sain,  “A  Multiobjective  Cost  Cumulant  Control  Problem:  The 
Nash  Game  Solution,”  Proceedings  American  Control  Conference,  Portland,  Oregon,  June 
8-10,2005. 

(b)  Manuscripts  submitted  to  Peer-Reviewed  Journals 

❖  Chang-Hee  Won,  “State  Feedback  Statistical  Control  and  Hamilton-Jacobi-Bellman 
Equations  for  a  Quasi-Linear  Stochastic  System,”  IEEE  Transactions  on  Automatic 
Control,  submitted. 

(c)  Manuscripts  in  Preparation  for  Peer-Reviewed  Journals 

❖  Chang-Hee  Won,  “Characteristics  of  Linear  Statistical  Control:  Bode  Integral,  Entropy, 
and  Risk-Sensitivity.” 

❖  Chang-Hee  Won,  Saroj  Biswas,  and  Randal  Beard,  “An  Algebraic  Method  of  Solving 
State-Feedback  Optimal  Controllers  for  a  Quasi-Linear  System  with  Quasi-Quadratic 
Cost.” 
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